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5 A VIRTUAL CONTROL PANEL 

FIELD OF THE INVENTION 

The present invention relates to a system and a method that en- 
10 ables a user to interact with a virtual control panel using a user 
controlled pointing object. The system comprises a first tracking 
unit adapted to capture data representing the position of the 
user controlled pointing object. 

15 The virtual control panel represents an interface for real-time 
human-system interactions. The invention can be applied for dif- 
ferent industrial processes where a user needs a control panel 
for interacting with a device or system, hereinafter referred to 
collectively as a device, regarding for example real-time data, 

20 historical data, simulations, test data, etc. The invention is par- 
ticularly useful in automation applications. 

PRIOR ART 

25 A field operator, e.g. in process plants, manufacturing, at off- 
shore platforms and substations, need both to interact with sys- 
tems or devices and to have access to information and docu- 
mentations. Various user interfaces, both physical and software 
interfaces, towards devices and system are often differently de- 

30 signed and the field operators have to learn to use every single 
user interface. The different interfaces often are very inconsis- 
tent. In addition to the two-way interaction with devices or sys- 
tems, the field operators need access to documentation, such as 
manual, historical data, maintenance & repair reports. Even 

35 though the trend is that such documentation becomes electroni- 
cally available, it often resides at different systems and loca- 




tions. Therefore, the field operators have to plan their work 
tasks in advance, or they even have to interrupt ongoing tasks in 
order to control the process or devices and find supportive in- 
formation and documentation. 

5 

Within manufacturing, a production line includes several robots, 
which are controlled by separate controllers and teach pen- 
dants. The operator need to interact with the robots, for exam- 
ple, in order to check status, inspect the robots and make new 
10 programs. A general drawback is that the operator must change 
Interaction device for each new robot even though several ro- 
bots may perform a task together. 

Augmented Reality (AR) is a method of overlaying real world 
15 representations with computer-generated graphics. Ideally, for 
vision-based augmented reality, the user will not be able to rec- 
ognize the difference between the real and the computer- 
generated graphics, and thereby the user will get an improved 
perception of the real world environment. Today, augmented re- 
20 ality techniques are used in a number of applications. Examples 
of use are within media, for example weather reporting, medi- 
cine, for example visualization of internal organs, for collabora- 
tive environments, for example virtual meeting rooms, and in 
process industries for maintenance and service. 

25 

Handheld and wearable control panels with interaction possibili- 
ties already exist and the interest of such devices increases. 
The use of mobile phones and PDAs as the interface towards 
systems or devices is known. Also, tablet PCs, which are used, 
30 e.g. within hospitals, provide an interface that the user easily 
carries around and interacts with by touching the screen. 

The newest field that is related to the invention is virtual key- 
boards. A virtual keyboard is projected, e.g. at a table, and will 
35 typically be the input mean for a PDA. The user touches or 
presses the keys and the system recognizes the specific 




touched keys like a standard keyboard. Virtual keyboards are 
commercial products offered by several manufacturers, e.g. 
Canesta, Senseboard Technologies and Samsung. 

5 US 6.614,422 discloses a digital user input to a companion sys- 
tem such as a PDA, a cell telephone or an appliance device, us- 
ing a virtual input device such as an Image of a keyboard. A 
sensor captures three-dimensional positional information as to 
location of the user's fingers in relation to where keys would be 

10 on an actual keyboard. This information is processed with re- 
spect to finger locations and velocities and shape to determine 
when a virtual key has been struck. The processed digital infor- 
mation is output to the companion system. 

15 US 6,618,425 disclose a virtual laser operator. A laser controller 
interconnected with an electric discharge laser communicates 
with a remote computer Incorporating a display screen prp- 
grammably emulating a conventional keyboard. The display 
screen has a plurality of imaged virtual keys each programmably 

20 emulating a physical key of a conventional keyboard. A key- 
stroke is typically applied by manually pressing the position of a 
corresponding virtual key on a touch sensitive screen, or alter- 
natively by actuating a conventional pointing device. 

25 The Issue of Interacting with virtual graphics has been ad- 
dressed, for example. In the film "Minority report". There is a 
scene where Tom Cruise uses his hands to interact with virtual 
Information visualized on a Plexiglas "wall". He further uses ges- 
tures for the interaction. 

30 

OBJECTS AND SUMMARY OF THE INVENTION 

The object of the present Invention is to provide a portable con- 
trol panel that the user can carry with him and which weighs al- 
35 most nothing. 



This object is achieved by means of the initially defined system, 
characterized in that the system further comprises: a portable 
identification element, a second traci<ing unit adapted to capture 
data representing the position of the identification element, a 
storage unit, storing at least one pre-defined graphical interface 
representing a control panel of a device, said graphical interface 
comprising an interface for user Interactions with the device, a 
graphics unit, generating a graphical representation of the con- 
trol panel based on said stored graphical Interface, a registering 
unit, registering said graphical representation of the control 
panel In a fixed relation to said portable Identification element, 
based on said data representing the position of the identification 
element, to produce a virtual control panel, a display unit, show- 
ing the user a view comprising the real world and the virtual 
control panel projected In a fixed relation to said portable identi- 
fication element, and an application unit, performing actions In 
response to the users interactions with the virtual control panel, 
and determining which actions to be performed based on the 
position of said user controlled pointing object in relation to the 
Identification element. With a device is meant either a single de- 
vice or a system comprising several devices. A virtual control 
panel is an augmented reality view of a real control panel. How- 
ever, the virtual control panel may not necessarily have the 
same look as conventional control panels. 

The user interacts with the virtual control panel using a pointing 
object. The pointing object Is for example a handheld pointing 
device or a part of the user's body, e.g. a hand or a finger. 

Preferably, said portable identification element is adapted to be 
carried by the user. The user may for example carry the identifi- 
cation element in his hand or attached to his body, e.g. attached 
to his arm or to a watch. If the user uses his finger as a pointing 
device and carries the identification element attached to the 
body, the user will have both his hands free to use for other 
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purposes. A portable identification element can be put aside, 
e.g. on a table while the user performs other tasks. 

The virtual control panel is Implemented as augmented reality 
5 (AR), which is computer-generated graphics combined with the 
real world. The field operator "carries" the virtual control panel 
with him. The virtual control panel Is attached to the identifica- 
tion element, which Is recognized by the second tracking unit. 
The virtual control panel only exists as data projections. The vir- 
10 tual control panel represents the user interface for Interacting 
with process devices and/or systems. The virtual control panel 
comprises for example Interaction buttons, menus and input 
means for set point changing. 

15 The low weight of the virtual control panel, i.e. the weight of the 
Identification element, and the fact that it can be attached to the 
body and Is wireless, are clear advantages of the virtual control 
panel according to the Invention. 

20 According to an embodiment of the invention the system is 
adapted to modify the appearance of the virtual control panel in 
response to interactions between the user controlled pointing 
object and the virtual control panel. This embodiment makes it 
possible for the user to navigate in different views of the virtual 

25 control panel and to interact with the control panel, for example 
to scroll a view and look Into menus. 

• • • 

I According to an embodiment of the invention at least one of the 

h stored graphical interfaces comprises more than one graphical 

30 view to be displayed on the virtual control panel, and which of 
the views to be displayed is determined based upon the users 
actions. As the virtual control panel may not have the exact 

:: same look as a conventional control panel, a device may have 

"* more than one graphical views related to It. 

:r 35 



According to an embodiment of the invention said graphical in- 
terface is adapted to display data from the device, and the sys- 
tem is adapted to generate a graphical representation of the 
data and to display the data on the virtual control panel. Pref- 

5 erably. the data is displayed in response to interactions between 
the user controlled pointing object and the virtual control panel. 
Examples of data visualized on the virtual control panel are real- 
time data, status monitoring, reports and other textual docu- 
ments, historical data In form of trend curves and 3D slmula- 

10 tions. Some of the graphical views of the virtual control panel 
will contain online data for devices in the process that allows a 
two-way interaction between the virtual control panel and the 
related device. When such data is updated due to changes In 
the process or within the device, e.g. change of temperature or 

5 pressure, this will automatically be updated at the virtual control 
panel. On the other hand, the user may change, e.g. the set- 
point for a specific device from the virtual control panel, and that 
device will automatically change its set-point. 

According to an embodiment of the invention, the storage unit is 
adapted for storing a plurality of graphical interfaces, each rep- 
resenting a control panel of a particular device, that the system 
is adapted to generate and display a plurality of graphical repre- 
sentations of control panels for different devices based on said 
stored graphical interfaces of the devices, and that the system 
comprises means for identifying which of the stored control pan- 
els to be displayed. The representations of different virtual con- 
trol panels are defined in advance and stored in the storage 
unit, e.g. at a computer or a server. This embodiment provides 
the user with a control panel that can be used for interacting 
with a plurality of different devices or systems. The virtual con- 
trol panel adapts its appearance, interface, and content to a 
specific device, for example to the device that the user stands in 
front of, or closest to. The AR system visualizes the related vir- 
tual control panel, which may provide a range of different infor- 
mation and information presentations. 




According to an embodiment of the invention said means for 
identifying wliich of tiie stored control panels to foe displayed 
comprises a recognition unit for recognizing and identifying de* 
5 vices in the environment of the user, and the system is adapted 
to determine which of the stored control panels to be displayed 
based on which of the device is identified. For example, the rec* 
ognition unit is adapted to recognize and identify unique identifi- 
cation markings on the devices. For instance, the recognition 
10 unit is adapted to detect the identity of the closest device. The 
virtual control panel changes its look and functionality depend- 
ing on which device is recognized. 

According to an embodiment of the invention the system is ar- 
15 ranged so that it changes the virtual control panel displayed 
when another device is recognized and identified. Thus, when 
the user moves towards another device, the virtual control panel 
change its look to represent the functionality and characteristics 
of the new device when the user has confirmed that he wants 
20 the virtual control panel of the other device. There may be cases 
where the devices physically are close to each other and the 
user may not change the virtual control panel just because he 
takes a step aside, for example, to get a better view of the spe- 
cific device without having the intention to change the virtual 
25 control panel to the virtual control panel for the neighbour de- 
vice. 

According to an embodiment of the invention said display unit 
comprises a wearable display showing the user said view. For 
30 example, the field operator wears glasses or a head-mounted 
display, on which the computer-generated graphics are pro- 
jected either as optical see-through or as video see-through. 

A further object of the invention is to provide a method that en- 
35 ables a user to walk around and interact with a virtual control 




panel using a user controlled pointing object. This object is 
achieved by the method of the invention as defined in claim 12. 

According to an aspect of the invention, the object is achieved 
5 by a computer program directly loadable into the Internal mem- 
ory of the computer or a processor, comprising software code 
portions for performing the steps of the method according to the 
invention, when said program is run on a computer The com- 
puter program product is provided either on a computer readable 
10 medium or through a network, such as the Internet. 

According to another aspect of the invention, the object is 
achieved by a computer readable medium having a program re- 
corded thereon, when the program is to make a computer per- 
15 form the steps of the method according to the invention, and 
said program is run on the computer. 

Advantages gained by this invention is that the field operator 
always has the control panel with him, the easy standardization 

20 of the various user interfaces, the user only has to interact with 
a single interface which changes its representation, e.g. look, 
content and functionality, in dependence of the closest and con- 
firmed device, and that the virtual control panel weights almost 
nothing. The field operator can put the virtual control panel 

25 aside while he is doing other tasks, e.g. put it down on a table, 
the floor, etc. The virtual control panel Itself has no need for 
ruggedized or intrinsic safe certification. 

BRIEF DESCRIPTION OF THE DRAWINGS 

30 

The invention will now be explained more closely by the descrip- 
tion of different embodiments of the invention and with reference 
to the appended figures. 

35 Fig. 1 shows a system for providing a virtual control panel ac- 
cording to an embodiment of the invention. 
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Fig. 2 shows a block scheme over a system according to the 
invention. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
OF THE INVENTION 

This invention proposes a system and a method for providing a 
virtual control panel, which adapts its interface and content to a 
device and/or a system by using augmented reality. A system for 
generating an augmented reality representation of computer- 
generated graphical Information overlaid the real world can be 
configured in different ways. The two most likely ways of visual- 
izing augmented reality are Video see-through and Optical see- 
through. Video see-through needs a camera to capture a live 
video stream of the environment corresponding to the user's 
view of the world. The system combines computer-generated 
graphics with the live video stream and projects the combined 
augmented reality video onto the display device. The user will 
see the video with overlaid virtual information as If he was look- 
ing at the real world. For optical see-through, the computer- 
generated graphics are registered directly onto the display de- 
vice and follows the user's view of the real world. The virtual 
graphics are overlaid the real world without Including a video of 
It. The former solution is less demanding regarding the frame 
rate. 

Figure 1 shows an embodiment of a system according to the in- 
vention. The system comprises a handheld Interacting and point- 
ing device 1, a handheld or otherwise wearable Identification 
element 3 having a unique Identification (ID), a tracking unit 5 
for determining the position of the pointing device 1 and the po- 
sition of the identification element 3 In relation to a world coor- 
dinate system, a wearable display device 7 for visualizing aug- 
mented reality overlaid the view of the real world, identification 
markings 9a, 9b, having unique Ids, for placing at devices 11a, 




11b in the environment, and a recognition system for recogniz- 
ing tlie identification markings. Preferably, the tracldng unit also 
determines the orientation of the pointing device 1 and the iden- 
tification element 3. 

5 

A virtual control panel is an augmented reality view of a real 
control panel. In this embodiment video see-through is used to 
generate and present the virtual control panel in relation to the 
identification element 3. The identification element 3 defines the 
10 position and orientation of the virtual control panel. The identifi- 
cation element is for example, a piece of paper or plastic pro- 
vided with a unique ID such as a bar code or any other identifi- 
cation means. 

15 The virtual control panel is visualized at the wearable display 
device 7. The wearable display device 7 is for Instance glasses, 
a head-mounted display or a head-up display. 

The system further comprises a camera 12 for capturing a 

20 stream of images of the environment, i.e. of the real world. The 
camera 12 is mounted in a fixed position relative to the display 
device 7, for example mounted on or integrated with the display 
device. The display device is located along the view axis of the 
camera and at the camera's Image plane. Thus, the camera is 

25 located in relation with the display In such a way that it provides 
the same view, as the user would get by looking directly at the 
real world. The live video stream of Images Is combined with 
computer-generated graphics representing the control panel, 
and the combination of the real world and the graphics is pre- 

30 sented in real-time at the display device 7. Additional functional- 
ity includes camera zooming with output of the actual camera 
focal length. This will enable the system to display the com- 
puter-generated graphics correctly while zooming. The camera 
12 may also be used for vision-based tracking by the tracking 

35 unit 5. 



Basically, two tracking units are needed for the virtual control 
panel system, In addition to a recognition system for recognizing 
the identification markings attached to the devices in the envi- 
ronment. The first tracking unit traces and determines the posi- 
tion, and possibly also the orientation, of the pointing device 1 , 
whereas the second tracking unit traces and determines the po- 
sition and orientation of the identification element 3 held by the 
user. These tracking units may be a single tracking unit used for 
tracing both, as In the embodiment shown in figure 1, or two dif- 
ferent tracking units. Examples of other possible tracking meth- 
ods and systems are ultrasonic tracking systems, magnetic 
tracking systems, tracking systems based upon Inertlal naviga- 
tion (using accelerometers and gyros), mechanical arms, optical 
systems and hybrid systems combining technologies previously 
mentioned. For some of the tracking systems, the pointing de- 
vice 3 and the identification element 3 may have integrated sen- 
sors for determining positions and orientations. Such sensors 
may, for example, be gyros, inertlal sensors and accelerome- 
ters. 

In this embodiment the user uses the pointing device 1 for inter- 
acting with the virtual control panel and for pointing in the envi- 
ronment. The camera 12 is used as vision-based tracking of the 
pointing device 1. The pointing device 1 can also be used for 
additional tasks such as to drag the virtual control panel, or 
other virtual information, from the identification element 3 to a 
specific device and to locate, select, show, etc. new positions in 
the environment. 

The embodiment disclosed in figure 1 uses vision-based track- 
ing. The tracking unit 5 utilizes the camera 12 for the vision- 
based tracking. The tracking unit 5 also includes a tracking 
module 15 comprising software for performing the tracking. The 
position and orientation of the pointing device 1 and the identifi- 
cation element 3 are determined by means of image recognition 
and the tracking module 15 comprises image recognition algo- 



rfthms. The camera 12 and the tracking module 15 can also be 
used for specifying the world coordinate system. 

Further the system comprises a wearable computer 17 contain- 
ing the necessary software in order to generate one or a plural- 
ity of virtual control panels based upon the output from the 
tracking module 15 and the video stream from the camera 12. 
The wearable computer 17 also contains the necessary software 
needed to perform the desired task or process, e.g. two-ways 
communication with devices, real-time data managing and visu- 
alization. Further, the wearable computer 17 will generate the 
graphics, which will provide the augmented reality interface 
view. Finally, the wearable computer 17 will contain a storage 
unit In order to save, and restore previously saved, information 
for example predefined graphical interfaces to devices. In an- 
other embodiment a stationary computer, or a server/client solu- 
tion, could be used instead of the wearable computer. 

Communication of information between the wearable computer 
17 and the camera 12, tracking module 15, and the display de- 
vice 7 is done through a wired or wireless link. Depending on 
the type of tracking unit, the pose of the interacting and pointing 
device 1 can come from the pointing device itself, or from an ex- 
. ternal tracking unit. 

A user 19 holds the identification element 3 in his hand or wears 
it on his body, e.g. around his hand wrist. The user holds the 
pointing device 1 in his other hand. The user also wears the dis- 
play device 7 and the wearable computer 17. The user may walk 
around in the environment carrying the virtual control panel. The 
tracking unit 5 recognizes known identification markings 9a, 9b 
in the environment. In case the tracking unit is vision-based, the 
user wears a camera 12 attached to his head. The camera re- 
cords live video of the real world. 



A fixed world coordinate system needs to be defined. The origo 
of the world coordinate systen) is defined to correspond with the 
display device 7. As the camera 12 Is mounted on, or integrated 
with, the display device, the relation between the camera and 
the display device Is known. The system determines the position 
of the pointing device 1 and the identification element 3 In rela- 
tion to the display device 7. Hence, all pose information pro- 
vided by the tracl<ing unit is related to this world coordinate sys- 
tem. 

The system also needs to recognize and Identify the identifica- 
tion markings 9a, 9b on the devices In the environment. The 
identification markings are attached in advance to devices and 
systems, which the user wants to interact with. Different tech- 
nologies can be used for the recognition such as RF tagging, 
bar codes, ultrasound and GPS. Alternatively, a vision based 
tracking unit as described above can be used to recognize the 
Identification markings. It is also possible to recognize the de- 
vices themselves by natural feature recognition. 

In relation to each device, virtual interfaces need to be defined 
in the system. Also, potential communication between the virtual 
interfaces and the devices must be defined and set up in ad- 
vance. The tracking unit 5 determines the ID of the specific de- 
vices based on the ID of the identification markings 9b, 9a. The 
ID'S have corresponding virtual Interfaces, which are presented 
as the virtual control panels. 

When the user starts using the system, he walks around in the 
environment until he comes to a device, which he wants to In- 
teract with. There may be several ways for the system and the 
user to decide which device should be presented on the virtual 
control panel. The user moves the Identification element 3 next 
to the device until the system has recognized both the ID for the 
specific device and the Identification element ID within a prede- 
termined distance. Another way is to select the actual device 



from a dynamic list of all online devices, that Is devices within a 
certain distance of the user so that there is robust communica- 
tion with the device. The related virtual interface appears as a 
virtual control panel attached to the identification element. The 
user uses the Interaction and pointing device 1 to interact with 
the virtual control panel, which may include interaction buttons, 
menus and other types of interaction possibilities. The virtual 
control panel updates Its look and content corresponding to the 
user's choices. 

The virtual control panel changes Its look and content as the 
user either moves along to a new device, having an identifica- 
tion marking, or as the user interacts with the virtual control 
panel. Examples of virtual Interfaces are device overviews, 
status information, set points, trend curves, documentation, 
video clips, e.g. for maintenance and procedures, and data en- 
try. The virtual control panel can be combined with other interac- 
tion modalities such as voice and tactile feedback. Typically! the 
virtual control panel may give an audio and/or a visual feedback 
when the user pushes an Interaction button. 

The user can move the identification element freely, both re- 
garding position and orientation, and the virtual control panel 
will follow the identification element as if it Is physically attached 
to the element. The related tracking unit captures the move- 
ments of the identification element and transmits its position and 
orientation to the wearable computer by means of a communica- 
tion link, e.g. wireless communication. Hence, the user can put 
the virtual control panel away, e.g. in a pocket when he is not 
using it, or put the virtual control panel on a table, etc. when do- 
ing other tasks. 

The wearable computer 17 communicates by wireless communi- 
cation with the devices, e.g. to retrieve and send data. Readings 
from the devices may be presented on the virtual control panel 
whereas the user may modify, e.g. the set point of a device. 



Figure 2 shows a block scheme over the system. The tracking 
unit 15 comprises a first tracking unit 16a for tracking the posi- 
tion and orientation of the pointing device 1 , and a second track- 
ing unit 15b for tracking the position and orientation of the iden- 
tification element 3. The tracking units 15a. 15b are adapted to 
determine the position and orientation of the pointing device 1 
and the identification element 3 by means of Image recognition, 
based on the images from the camera 12. Output from the track- 
ing module 16a Is the position and orientation of the pointing 
device 1, and output from the tracking module 15b Is the posi- 
tion and orientation the identification element, which outputs are 
transferred to an application unit 21. 

The application unit 21 contains the necessary software in order 
to perform the desired process or task, e.g. integrate real-time 
data with the recognized device, and perform actions based on 
input from the pointing device 1. Which action to be performed 
depends on the position of the pointing device relative to the 
identification element and on stored functionality of the control 
panel. Further, the application unit 21 holds Information regard- 
ing the world coordinate system. Hence, the application unit 21 
Is dependent on the specific system and environment. 

A graphics unit 23 generates a 3D graphical representation of 
the visual information that is to be displayed in relation to the 
identification element 3. That is, this unit holds a specification of 
the 3D graphical primitives to be visualized. The graphics unit 
23 receives information from the application unit 21 regarding 
the virtual graphical Information to be displayed based on user 
Input, stored Information and Information received from the con- 
trolled device. All graphical primitives positions are specified in 
relation to the world coordinate system. For a robot program- 
ming application relevant visual Information may be user- 
specified waypolnts, the actual robot path, task specific informa- 
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tion etc. The graphical representations produced in the graphics 
unit 23 are transmitted to a registering unit 25. 

The camera 12 produces Jive video real world Images, which are 
Input to the registering unit 25. The position and orientation of 
the camera and display Is determined according to the world co- 
ordmate system, and is used by the registering unit 25 in order 
to overlay the real world scene with computer-generated graph- 
ics from the graphics unit 23. The registering unit 25 combines 
the virtual graphics from the graphics unit 23 with video Images 
from the camera 12. With the registration correctly done, the 
computer-generated graphics will be virtually "attached" to the 
real world scene. The position and orientation of the identifica- 
tion element is transferred from the tracking unit 15b to the reg- 
is istering unit. The computer-generated graphics represents the 
virtual control panel and Its location In relation to the real world 
IS determined based on the position and orientation of the Iden- 
tification element. The combined images are displayed on the 
display device 7. 

20 

A storage unit 29 enables saving and loading of application re- 
lated information, e.g. application related information, graphical 
interfaces for the various devices, and system configuration pa- 
rameters, e.g. local and world coordinate frame. The storage 
> unit IS also adapted to store.a plurality of pre-defined graphical 
interface representing control panels of devices and/or systems, 
which graphical Interfaces comprises interfaces for user Interac- 
tions with the device. 



In this embodiment, the camera 12 and a vision-based tracking 
system is used for detecting the Identification markings on the 
devices. A recognition unit 31 recognizes and Identifies devices 
m the environment of the user, based on output from the camera 
12. The recognition unit 31 recognizes the identification mark- 
ings on the devices, reads the ID on the marking and identifies 
the device by comparing the read ID with ID'S stored in advance. 



When a device is identified, the recognition unit 31 Informs the 
application unit 21 about which device is Identified, and then It 
is possible for the application unit to determine which of the 
stored control panels to be displayed. When a new device Is 
recognized and Identified, the Identity of the device is displayed 
to the user, who can choose between accepting the new device, 
or rejecting it. If the user accepts the new device, the applica- 
tion unit automatically retrieves the graphical Interface of the 
new identified device from the storage unit 29, and delivers the 
new graphical interface to the graphics unit 23. Thereby, the vir- 
tual control panel displayed is changed when a new device is 
identified, and when the user has confirmed the shift of the 
graphical interface. 

As the virtual control panel may not have exactly the same look 
as a conventional control panel, a device may have more than 
one graphical view related to it. Thus, the graphical interface 
may contain more than one graphical view. As an example, the 
first graphical view of the virtual control panel for a pump that 
meets the user may be an overview of the performance visual- 
ized as an analogue instrument Indicating the performance rang- 
ing from 0 to 100%. The specific performance information for the 
pump Is retrieved in real-time from the pump device via the 
computer and some communication means, preferably wireless, 
between the computer and the device, or via a server. 

The graphical view may further have interaction buttons for 
'on/off, 'Set-point' and 'Main menu'. These buttons may be pre- 
sented as 3D objects on the graphical view. The 'on/off interac- 
tion button may be color-coded so that It, for example, is green 
when the pump is on and red when the pump is stopped. Addi- 
tionally, the text at the Interaction button may change between 
'On' and 'Off to Indicate the status. When the pump is running 
the status is 'on', the interaction button may look like It Is 
pressed, colored in green with the text 'On'. When the pump is 
stopped, the interaction button will have changed its color to 



red, with the text 'Off and the interaction button may lool< like it 
is un-pressed. 



The Interaction button 'Set-point' may bring up a different 
graphical view representing a view for change the set-point. The 
set-point may be visualized as a number in a display which can 
be selected. The user may choose between different ways of 
changing the set-point. One way is to select the displayed num- 
ber and enter a new number for the new set-point. Another way 
may be to adjust the set-point using interaction buttons for 'up' 
and 'down'. The set-point will be updated immediately in the 
display. Regarding the way to change set-point, the user may 
confipm the new set-point before the device is updated (in real- 
time). 

The main menu may. for example, contain options for 'Perform- 
ance', 'Set-point', 'Documentation'. 'Trend curves', and 'Mainte- 
nance history'. These options may, for example, be presented 
as interaction buttons, menus or tabs. 'Documentation' may In- 
clude documentation for various user groups and usages, for 
example, user manuals for the device, procedures, technical 
manuals, reports, etc. When the user has selected the type of 
documentation. It may be shown page by page where the user 
can select 'next' and 'previous', e.g. by interaction buttons. As a 
part of documentation, pictures, graphics and videos may also 
be included. 



The graphical views are designed and developed beforehand 
and stored in the storage unit. In relation with the graphical vi- 
ews, the position of and content of Interaction field on the 
graphical views are also stored, similarly are the data fields 
stored. When a graphical view is uploaded related to the virtual 
control panel, data Is automatically filled In the data field. User 
interactions with the virtual control panel are taking care of by 
the tracking system, which traces the pointing object's position. 
The application unit notices when the position of the pointing 



object matches any of the interaction fields at the specific 
graphical view of the virtual control panel. 

In one use-case, a user working within a plant needs to check 
and update an existing robot program to make sure that the pro- 
gram modules are downloaded and related to the right robots. 
Four robots are Included in the program. He then has to log in at 
each of the four teach pendants, one at a time, scroll down in 
the menus and find the right program and the right place In the 
program. By utilizing the proposed invention, the user would 
only need to walk next to the appropriate robot and then check 
the program for each of the four robots. Costs related to main- 
taining the teach pendants will be reduced. Also, the problem 
with wires between the robot controller and the teach pendant is 
avoided with the virtual control panel. 

In process plant environments the user walks around and per- 
forms monitoring, inspection and maintenance tasks related to a 
process or device. The user typically has two-way radio commu- 
nication with the control room users to transfer information to 
and from the process. The virtual control panel provides the field 
user with an interface for online interacting with the various de- 
vices and systems In the field as well as a tool for viewing 
documentation, reports, maintenance and service procedures, 
etc. The user always carries the virtual control panel with him. 
He can put It aside, or attach It. e.g. to his arm, while doing 
other tasks. The field user will be able to perform his tasks more 
efficiently. Also, the use of the virtual control panel prevents er- 
rors due to communication problems between the field user and 
the control room (misunderstandings, noise, etc.). The control 
room users may also have the opportunity to see updated data 
and information from the plant. 

The virtual control panel can also be used for other applications 
such as registration and counting in stores. Combined with bar 
codes, the virtual control. panel may visualize Information related 
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to the registered part. The user can enter data and Information 
as well as reading status and documentation. 

Another example application is an office. The virtual control pa- 
nel may provide the user with names of people by passing the 
doors, or the people themselves if the employees are equipped 
with IDs. For example, the security staff may have virtual control 
panels for checking IDs, changing passwords, recording status 
information from their check rounds, etc. 

The present Invention is not limited to the embodiments dls- 
closed, but may be varied and modified within the scope of the 
following claims. For example a device could be identified by the 
user Inputting an Identification number to the system. 

Alternatively, the user may use his hand and fingers for Interact- 
ing with the virtual control panel. The system still needs to de- 
termine the pose of the interaction body part. A similar tracking 
unit, as for the Interacting and pointing device, can be used for 
hand/finger tracking. The system further needs a recognition 
system in order to recognize the interacting body part and de- 
termine its position. 

In addition to using his fingers or a pointing device, the field 
user may interact with the virtual control panel using other Inter- 
action means such as voice and physical interaction buttons. 

For example, the present system is not limited to augmented re- 
ality systems utilizing a composed video image but may also be 
utilized in see-through augmented reality systems, in which only 
the computer generated graphics are presented to the user who 
views the graphics with the real world In the background. 

In another embodiment, instead of having a display device, the 
display unit is arranged so that the view is directly projected on 
the eyes of the user. For example, a virtual retinal display, 



where information is projected directly Into the retina without the 
need of a physical display device, could be used. 



CLAIMS 



1 . A system that enables a user to interact with a virtual con- 
trol panel using a user controlled pointing object (1), wherein 
the system comprises a first tracking unit (15b) adapted to cap- 
ture data representing the position of the pointing object, char- 
acterized in that the system further comprises: 

- a portable identification element (3), 

- a second tracl<lng unit (15a) adapted to capture data repre- 

senting the position of the identification element, 

- a storage unit (29), storing at least one pre-defined graphi- 

cal interface representing a control panel of a device, 
said graphical interface comprising an interface for user 
interactions with the device, 

- a graphics unit (23), generating a graphical representation 

of the control panel based on said stored graphical inter- 
face, 

- a registering unit (25), registering said graphical represen- 

tation of the control panel in a fixed relation to said port- 
able identification element, based on said data repre- 
senting the position of the identification element, to pro- 
duce a virtual control panel, 

- a display unit (7), showing the user a view comprising the 

real world and the virtual control panel projected in a 
fixed relation to said portable identification element, and 

- an application unit (21), performing actions in response to 

the users interactions with the virtual control panel, and 
determining which actions to be performed based on the 
position of said user controlled pointing object In relation 
to the identification elenient. 

2. A system according to claim 1, characterized in t hat the 
system is adapted to modify the appearance of the virtual con- 
trol panel in response to Interactions between the user con- 
trolled pointing object and the virtual control panel. 
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3. A system according to claim 1 or 2, characterized In that 
said graphical interface is adapted to display data from the de- 
vice and that the system is adapted to generate a graphical rep- 
resentation of the data and to display the data on the virtual 
control panel. 

4. A system according to any of the previous claims, charac- 
terized in that said user controlled pointing object is a handheld 
pointing device (1) or a part of the user's body. 



5. A system according to any of the previous claims, charac- 
terized in that the storage unit (29) is adapted to store a plural- 
ity of graphical interfaces, each representing a control panel of 
a particular device, that the system is adapted to generate and 
15 display a plurality of graphical representations of control panels 
for different devices based on said stored graphical interfaces of 
the devices, and that the system comprises means (31) for iden- 
tifying which of the stored control panels to be displayed. 

20 6. A system according to claim 5, characterized in that said 
means for identifying which of the stored control panels to be 
displayed comprises a recognition unit (31) for recognizing and 
identifying devices in the environment of the user, and that the 
system Is adapted to determine which of the stored control pan- 

25 els to be displayed based on which of the devices is identified. 

7, A system according to claim 6, characterized in that said 
recognition unit (31) is adapted to recognize and identify unique 
identification markings (9a, 9b) on the devices (11a, lib). 



8. A system according to any of the claims 5-7, characterized 
in that the system is arranged so that it changes the virtual con- 
trol panel displayed when another device is recognized and 
identified, and when the user has accepted the device. 
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9. A system according to any of the previous claims, charac- 
terized in that said portable identification element (3) is adapted 
to be carried by the user during interaction with the virtual con- 
trol panel. 

5 

10. A system according to any of the previous claims, charac- 
terized In that said portable identification element (3) is attach- 
able to the body of the user 

10 11. A system according to any of the previous claims, charac- 
terized in that said display unit comprises a wearable display 
device (7) showing the user said view. 

12. A method that enables a user to interact with a virtual con- 
15 trol panel using a user controlled pointing object, wherein the 
method comprises: 

- receiving data representing the position of the user con- 

trolled pointing object (1), 

- receiving data representing the position of a portable identi- 
20 fication element (3), 

- storing at least one pre-defined graphical interface repre- 

senting a control panel of a device, 

- generating a graphical representation of the control panel of 

the device based on said pre-defined graphical Interface, 
25 - registering said graphical representation of the control 

panel in a fixed relation to said portable identification 
element, based on said data representing the position of 
the identification element, to produce a virtual control 
panel, 

30 - displaying a view comprising the real world and the virtual 

control panel projected in a fixed relation to said portable 
identification element, and 

- performing actions in response to the users interactions 

with the virtual control panel, wherein the actions to be 
35 performed is determined based on the position of said 
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user controlled pointing object in relation to the position 
of the virtual control panel. 

13. A method according to claim 12, comprising modifying the 
5 appearance of the virtual control panel in response to interac- 
tions between the user controlled pointing object and the virtual 
control panel. 

14. A method according to claim 12 or 13, wherein it comprises, 
10 defining a two-way communication between the virtual control 

panel and the device, sending information to the device regard- 
ing the users actions with the virtual control panel, receiving 
data from the device, generating a graphical representation of 
the received data and displaying the data on the virtual control 
15 panel. 

15. A method according to claim 14, wherein said data is dis- 
played on the virtual control panel in response to interactions 
between the user controlled pointing object and the virtual con- 

20 trol panel. 

16. A method according to any of the claims 12 - 15, wherein 
the method comprises: 

- storing a plurality of pre-defined graphical Interfaces, each 
25 representing a control panel of a particular device, 

- determining which of the stored control panels to be dis- 

played, and 

- generating a graphical representation of the control panel to 

be displayed based on the pre-defined graphical inter- 
30 face of the control panel to be displayed. 

17. A method according to any of the claims 12 - 16, wherein at 
least one of the stored graphical interfaces comprises more than 
one graphical view to be displayed on the virtual control panel, 

35 and which of the views to be displayed is determined based 
upon the users actions. 




18. A method according to claim 16, wherein it comprises rec- 
ognizing and Identifying a device, determining which of the 
stored control panels to be displayed based on the identified 

5 device, and generating a graphical representation of the control 
panel of the identified device based on the stored graphical in- 
terface of the identified device and displaying a view comprising 
the real world and the virtual control panel of the Identified de- 
vice projected in a fixed relation to said portable identification 
10 element. 

19. A method according to claim 18 characterized in that each 
device is provided with a unique identification marking and a de- 
vice is recognized by identifying its unique identification mark- 

15 Ing. 

20. A method according to any of the claims 18 - 19, wherein 
the virtual control panel displayed is changed when another de- 
vice is recognized and identified, and when the user has ac- 

20 cepted the device. 

21. A method according to any of the claims 12 - 20, wherein 
said portable identification element is carried by the user during 
interaction with the virtual control panel. 

25 

22. A method according to any of the claims 12 - 21, wherein 
the virtual control panel comprises virtual interaction members 

: and an audio and/or visual feedback is generated when the user 

■l' activates any of the virtual interaction members. 

• ■ • ■ 

" : 30 

23. A computer program comprising program instructions which, 
when loaded into a computer, causes the computer to perform 
the process of any of the claims 12-22. 

* A « 

"']' 35 24. A computer readable medium having a program recorded 

thereon, where the program is to make a computer perform the 



steps of any of the claims 12 -22, when said program is run 
the computer. 
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ABSTRACT 

A system that enables a user to interact with a virtual control 
5 panel using a user controlled pointing object (1). The system 
comprises a portable identification element (3), a tracking unit 
(15) adapted to capture data representing the position of the 
pointing object and the position of the identification element, a 
storage unit, storing at least one pre-defined graphical interface 

10 representing a control panel of a device, a graphics unit, gener- 
ating a graphical representation of the control panel, a register- 
ing unit, registering said graphical representation of the control 
panel in a fixed relation to said portable identification element to 
produce a virtual control panel and a display unit (7), showing 

15 the user a view comprising the real world and the virtual control 
panel. 

(Figure 1) 
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